Stereoscopic image encoding

ABSTRACT

There is disclosed a method for storing and/or transmitting 3D image information comprising the steps of: producing an image to be stored and/or transmitted comprising an array of strongly correlated neighbouring sub-images; casting the sub-images on to a pixel screen capturing the sub-images as electronic data; compressing the electronic data by eliminating redundancies associated with the sub-images; storing and/or transmitting the compressed data; the compression being reversible so as to expand the data to re-create the sub-images for viewing as a 3D image through an optical viewing system comprising a microlens or lenticular array.

[0001] This invention relates to storing and/or transmitting 3D images.3D images can be formed in a variety of ways. No matter how they areformed, there is substantially more information content in a 3D imagethan in a corresponding 2D image—the depth information is additional.Storing and/or transmitting 3D images therefore is more demanding ofstorage space or bandwidth than for 2D images, much as colour images aremore demanding than monochrome images. Coloured 3D images would appearon the face of it to be very demanding, but the problems can be eased bydata compression techniques, surprisingly to such a degree as brings 3Dtelevision into immediate prospect.

[0002] Methods for making (and viewing) 3D images—autostereoscopicimages, i.e. not requiring aids such as spectacles to view—are known andinvolve the use of an optical imaging system comprising a microlensarray of small spherical or lenticular (i.e. cylindrical) lenses. Suchimaging techniques produce images which are particularly well adapted,as it turns out, to compression, and the present invention isparticularly concerned with such imaging techniques.

[0003] The invention comprises a method for storing and/or transmitting3D image information comprising the steps of:

[0004] producing an image to be stored and/or transmitted comprising anarray of strongly correlated neighbouring sub-images;

[0005] casting the sub-images on to a pixel screen capturing thesub-images as electronic data;

[0006] compressing the electronic data by eliminating redundanciesassociated with the sub-images;

[0007] storing and/or transmitting the compressed data;

[0008] the compression being reversible so as to expand the data tore-create the sub-images for viewing as a 3-D image through an opticalviewing system comprising a microlens or lenticular array.

[0009] The image may be of a scene and produced using an optical imagingsystem comprising a microlens or lenticular array of small spherical orcylindrical lenses each of which images the scene from a slightlydifferent viewpoint.

[0010] The image may however be electronically generated or partiallyelectronically generated. Photographic images may be electronicallyscanned and captured as electronic data.

[0011] Small sub-image data sectors generated by the optical system arefed successively into an encoder where a previously fed sub-image issubstracted from the most recently fed-in sub-image by a differentialpulse code modulation (D P C M) coding technique to remove redundanciesbetween the sub-images.

[0012] Redundancies may be eliminated within the sub-images themselvesby techniques for example normally used in compression of twodimensional image data such for example as a discrete cosine transform(DCT) coding scheme.

[0013] A 3D-DCT coding scheme may be applied directly to groups ofsub-images, the use of the third transform dimension eliminatinginter-sub-group redundancies with the first two transform dimensionsused to remove intra-sub-image redundancies.

[0014] A quantisation function may be applied to the coded data thatsets small values to zero and transforms all other non-zero values tonearest values in a set of preferred values.

[0015] The coded data may then be entropy encoded.

[0016] The above coding schemes are suitable for compressing still imagedata. For storing and/or transmitting moving 3D image information aDPCM/3D-DCT coding scheme may be used, the DPCM coding decorrelatingimage data in the temporal domain and the 3D-DCT scheme eliminatingspatial redundancies.

[0017] A hybrid DPCM2/DCT scheme may be used for compression of moving3D image information, in which a 2D-DCT scheme decorrelates and henceremoves redundancies within each sub-image and two DPCM loops are used,one to remove redundancies between sub-images in a spatial sense whilethe second is used to remove temporal (inter-frame) redundancies.

[0018] Both of these moving 3D-image compression schemes may make use ofmotion compensation to achieve greater overall image reduction.

[0019] Methods for storing and/or transmitting 3D image informationaccording to the invention will now be described with reference to theaccompanying drawings, in which:

[0020]FIG. 1 is a diagrammatic illustration of an optical system castingan image on an electronic imaging device and display arrangements forimages therefrom;

[0021]FIG. 2 is a diagrammatic illustration of the imaging process inthe optical system of FIG. 1;

[0022]FIG. 3 is a section of a lenticular-integral image;

[0023]FIG. 4 is a section of a full-integral image;

[0024]FIG. 5 is a diagrammatic illustration of a first coding scheme forstill 3D image data compression;

[0025]FIG. 6 is a diagrammatic illustration of a second coding schemefor still 3D image data compression;

[0026]FIG. 7 is a diagrammatic illustration of a coding scheme formoving 3D image data compression;

[0027]FIG. 8 is a diagrammatic illustration of a scheme for an inputstructure for the coding schemes of FIGS. 5 to 7; and

[0028]FIG. 9 is a diagrammatic illustration of a scanning strategy usedin the coding schemes of FIGS. 5 to 7.

[0029] The drawings illustrate methods for storing and/or transmitting(and, of course, displaying or replaying) 3D image information.

[0030]FIGS. 1 and 2 illustrate imaging a scene S to be stored and/ortransmitted using an optical imaging system 11 comprising a microlens orlenticular array 12 of small spherical or lenticular (i.e. cylindrical)lenses each of which images the scene S from a slightly differentviewpoint to produce an array of strongly correlated sub-images.

[0031] The optical imaging system 11 comprises a front-end opticalarrangement 13 comprising a segmented wide aperture input lens 14, amicrotelescopic array 15 (a double integral, autocollimating microlensarray, see FIG. 2), and a segmented output macrolens array 16.

[0032] As seen in FIG. 2, each segment of the input lens array 14individually transposes its image at the focusing screen 17 of the array15 (see FIG. 2). The screen 17 comprises a double microlens screen. Eachsegment of the output lens array 16 projects the transposed images to besuperposed at a particular plane. The initial transposition andprojection will also produce a reversal of parallax between theindividual superposed image fields. As the initial transposed image isformed on the double integral microlens screen 17 it is presented to theoutput lenses as a spatially reversed 3D optical model. The resultingsuperposed 3D image is constructed from the integration of all thespatially reversed optical models projected by each lens segment, andconsequently continuity of parallax throughout the viewing angle isachieved.

[0033] The recorded image is a planar 2D image, which contains all the3D information relating to the scene S. This image, cast on themicrolens encoding screen 12, is reduced in size by a copy lensarrangement 19 to form a reduced image on an electronic image capturedevice such as a high resolution CCD array 21. The same can be achievedby imaging directly on to a high resolution CCD array which is overlayedwith a microlens encoding system.

[0034]FIG. 1 also illustrates image viewing arrangements, namely a flatpanel display 22 and a projection display 23 for viewing the imagecaptured by the device 21 as a 3D image. The flat panel display 22comprises a high density pixel screen 24 (which may be a liquid crystaldisplay panel or a cathode ray tube or a gas plasma screen) with anadjacent lens array 25 which acts as a decoding screen to decode thecoded information produced by the encoding screen 12.

[0035] The projection arrangement 23 comprises a high resolution videoscreen 26 and a projection lens arrangement 27 projecting the codedimage on the screen 26 on to an integral back projection decoding screen28 by which the observer sees a 3D image. A light valve/LCD arrangementmay be used in place of a video screen.

[0036]FIGS. 3 and 4 show sections (magnified) of coded images producedby arrangements such as that illustrated in FIGS. 1 and 2. Instead of anintegral coding arrangement, using spherical microlenses, a lenticulararrangement can be used, using cylindrical lenses, the lenses beingarranged with their axes vertical to correspond to the horizontalspacing of the eyes in binocular vision. FIG. 3 shows a 64×64 pixelsection of a lenticular-integral image, while FIG. 4 shows sixsub-images of a full-integral image each of 8×8 pixels.

[0037] It is with the recording, transmission, storage, reception,retrieval and display of such coded images as are seen in FIGS. 3 and 4that this invention is concerned.

[0038] The minimum bandwidth initially believed to be required for thetransmission of moving integral 3D images was 42 GHz. In fact, fullcolour 3D display is possible using a receiver with a bandwidth nogreater than is required for HDTV. A compression ratio of approximately4:1 is required for the transmission of integral 3D TV pictures. Highercompression rates than 4:1 are possible, enabling higher quality display(more pixels per sub-image) and more efficient use of transmissionbandwidth or storage space, i.e. allowing several TV picture channels tohave the same broadcast channel.

[0039] Conventional compression algorithms for HDTV, such as transformand sub-band coding techniques, achieve compression by decorrelating inthe spatial and/or temporal correlation domains. A totally white screen,of course, is totally correlated, while a picture with areas of more orless solid colour is less, but still fairly well correlated spatially.Successive frames of a television transmission are usually very wellcorrelated temporally, even with high-action scenarios, and there issubstantial scope for compression of ordinary 2D colour pictureinformation on all those accounts.

[0040] The addition of the third spatial dimension would appear to posesubstantial problems, but the invention overcomes the perceiveddifficulties by working with the strongly correlated—as will be evidentfrom FIGS. 3 and 4—sub-images, i.e. portions of the full image producedby the optical system described with reference to FIGS. 1 and 2.

[0041] The invention comprises compressing the electronic data producedby the CCD array 21 (or other electronic imaging device) by eliminatingredundancies between these strongly correlated sub-images before storingor transmitting the compressed data.

[0042] The compression, further according to the invention, isreversible so as to expand the data to recreate the sub-images forviewing as a 3D image through an optical viewing system such as eitherof the systems A, B of FIG. 1.

[0043]FIGS. 5 and 6 illustrate two still 3D image coding schemes,utilising two different decorrelation techniques namely:

[0044] small sub-image data sectors (such as the six sectors seen inFIG. 4) generated by the optical system are fed successively into anencoder where a previously coded sub-image is subtracted from the mostrecently fed-in sub-image by a differential pulse code modulation (DPCM)technique, and

[0045] intra-sub-image redundancies are eliminated using a discretecosine transform (DCT) technique.

[0046]FIG. 5, which shows both encoder and decoder (as do FIGS. 6 and 7)illustrates a hybrid DPCM/DCT coding scheme in which sub-images areinput (as electronic data representing colour/brightness values) into aloop 51 in which a previous sub-image is held in a store 52 to be fed toa subtraction unit 53—the DPCM step. The resulting partiallydecorrelated sub-image is fed to a DCT stage 54 where intra-sub-imageredundancies are eliminated.

[0047] The thus further decorrelated sub-image data then go to aquantiser 55 where all low value pixel values are set to zero and allothers are reduced to the nearest one of a small set of discrete values.The quantised data are then fed to an entropy encoder 56 that achievesfurther gain in compression by evaluating the statistics of theoccurrence of non-zero coefficient values and representing the quantiseddata with regard to their statistical significance is such a way as tominimise output data quantity.

[0048] The quantised data are fed, in the loop 51, to a de-quantiser 57that restores the pre-quantised values, then to an inverse DCT unit 58that effectively restores the sub-image to be passed to the store 52.

[0049] The final coded sub-image data is output from the entropy coder56.

[0050] The DCT coder 54 for intra-sub-image correlates applies theformula${F\left( {u,v} \right)} = {\frac{D_{u}D_{v}}{8}{\sum\limits_{l = o}^{7}{\sum\limits_{m = 0}^{7}{{f\left( {l,m} \right)}\cos \frac{\pi \quad {u\left( {{2l} + 1} \right)}}{16}\cos \frac{\pi \quad {v\left( {{2m} + 1} \right)}}{16}}}}}$

[0051] where

[0052] f(l,m) is the input data array formed from an 8×3 sub-imagedifference;

[0053] F (u,v) is the resulting transform coefficient array; and

D _(s)=1 if s=0, {square root}{square root over (2)} if S>0

[0054] The decoder illustrated in FIG. 5 comprises an entropy decoder 61receiving the input coded sub-images and passing them to a de-quantiser62 thence to an inverse DCT stage 63 and finally into a loop 64 with asub-image store 65 for the inverse DPCM stage, the restored sub-imagebeing output from the loop 64.

[0055] In between the coder and decoder, of course, will be a UHFtransmission of the compressed data and/or a storage on e.g. magnetic orvideo disc recording medium.

[0056]FIG. 6 shows a purely DCT encoding scheme comprising a 3D -DCTstage 66, quantiser 67 and entropy encoder 68 outputting the compresseddata. The decoder comprises the inverse elements, namely entropy decoder71, de-quantiser 72 and 3D inverse DCT stage 73.

[0057] The 3D -DCT stage for four 8×8 pixel sub-images is${F\left( {u,v,w} \right)} = {\frac{D_{u}D_{v}D_{w}}{16}\quad {\sum\limits_{l = 0}^{7}{\sum\limits_{m = 0}^{7}{\sum\limits_{7 = 0}^{3}{{f\left( {l,m,n} \right)}\cos \frac{\pi \quad {u\left( {{2m} + 1} \right)}}{16}\cos \frac{\pi \quad {v\left( {{2n} + 1} \right)}}{16}\cos \frac{\pi \quad {w\left( {{2l} + 1} \right)}}{16}}}}}}$

[0058] The third transform dimension takes account of inter-sub-imageredundancies such that a small group of sub-images is completelydecorrelated in a single transform calculation.

[0059] The arrangements discussed with reference to FIGS. 5 and 6 aresuitable for data compression for still pictures. An arrangement forcompression of moving integral 3D-TV pictures is essentially the same asthe arrangement of FIG. 5 except that the 2D DCT stage 54 is replaced bya 3D DCT stage and the 2D IDCT stages 58 and 63 are replaced by 3D IDCTstages.

[0060]FIG. 7 illustrates a further coding scheme for moving integral 3Dimages.

[0061] A DPCM stage 74 has a substractor 75 and sub-image store 76, asbefore. This passes DPCM decorrelated sub-images to a hybrid DPCM/DCTcoding arrangement 77 with a DCT coder 78, quantiser 79 and entropycoder 81 together with a DPCM loop 82 with dequantiser 83 and IDCT stage84, sub-image store 85 and, additionally, a motion compensation stage87. DPCM is used in this scheme to decorrelate in the temporal andinter-sub-image domains, DCT to decorrelate intra-sub-imageredundancies.

[0062] The decoder has an entropy decoder 87, a dequantiser 88 and IDCTstage 89 with two inverse DPCM loops 91, 92.

[0063]FIG. 8 illustrates, for 3D-DCT-based schemes, a strategy for theextraction of a group of sub-images from an integral image, thesub-images I, II, III, IV being extracted one after the other andassembled (notionally, as frame data) in the order to be fed to thecoding arrangement. Selection of groups of neighbouring sub-imagesmaximises the inter-sub-image correlation, allowing decorrelation toreduce the information required to be transmitted to a minimum.

[0064]FIG. 9 illustrates an entropy-coder scanning strategy for four 8×8pixel transformed and quantised sub-images. Each plane of the processedsub-image group is scanned in turn over the u and v axis directionsaccording to the scanning scheme illustrated in the right hand part ofthis figure. This zig-zag diagonal scanning statistically increases thezero run lengths leading to enhancement of compression by entropycoding.

[0065] Using the techniques described and illustrated, moving 3D colourpictures can be transmitted within a standard UHF terrestrial TVbandwidth, the images being true colour, unlike holography, and the 3Dimaging being integral, with no flipping over a wide viewing angle.

[0066] The system described is compatible with 2D television receiversinasmuch as without the optical decoding arrangements the picture isidentical to what the comparable 2D picture would be but would appearslightly out of focus. This can be compensated for by an imageenhancement technique to sharpen the focus.

1. A method for storing and/or transmitting 3D image informationcomprising the steps of: producing an image to be stored and/ortransmitted comprising an array of strongly correlated neighbouringsub-images; casting the sub-images on to a pixel screen capturing thesub-images as electronic data; compressing the electronic data byeliminating redundancies associated with the sub-images; storing and/ortransmitting the compressed data; the compression being reversible so asto expand the data to re-create the sub-images for viewing as a 3-Dimage through an optical viewing system comprising a microlens orlenticular array.
 2. A method according to claim 1 in which the image isof a scene produced using an optical imaging system comprising amicrolens or lenticular array of small spherical or cylindrical lenseseach of which images the scene from a slightly different viewpoint.
 3. Amethod according to claim 1 or claim 2, in which the image iselectronically generated or partially electronically generated.
 4. Amethod according to any one of claims 1 to 3, in which photographicimages are electronically scanned and captured as electronic data.
 5. Amethod according to any one of claims 1 to 4, in which redundanciesbetween sub-images are eliminated.
 6. A method according to any one ofclaims 1 to 5, in which redundancies within sub-images are eliminated.7. A method according to any one of claims 1 to 5, in which smallsub-image data sectors generated by the optical system are fedsuccessively into an encoder where a previously coded sub-image issubtracted from the most-recently fed-in sub-image by a differentialpulse code modulation (DPCM) coding technique.
 8. A method according toclaim 7, in which redundancies are eliminated within the sub-imagesthemselves.
 9. A method according to claim 8, in which theintra-sub-image redundancies are eliminated using a discrete cosinetransform (DCT) coding scheme.
 10. A method according to any one ofclaims 1 to 9, in which a 3D-DCT coding scheme is applied directly togroups of sub-images, the third dimension eliminating inter-sub-imageredundancies.
 11. A method according to any one of claims 1 to 10, inwhich a quantisation function is applied to the coded data that setssmall values to zero and transforms all other non-zero values to nearestvalues in a set of preferred values.
 12. A method according to any oneof claims 1 to 11, in which the coded data are further entropy encoded.13. A method according to any one of claims 1 to 12, for storing and/ortransmitting moving 3D image information using a DPCM coding techniquedecorrelating image data in the temporal domain and 3D-DCT eliminatingspatial redundancies.
 14. A method according to any one of claims 1 to13, in which a 2D-DCT scheme correlates intra-sub-image spatialredundancies and a DPCM technique decorrelates inter-sub-image data bothin the spatial and temporal domains.